{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eNbQXYWNj2U9"
   },
   "source": [
    "# Hindi and Tamil Question Answer / RAG\n",
    "\n",
    "In this notebook, we use new Navarasa LLMs from TeluguLLM to create a Hindi and Tamil Question Answering system. Since we're using a 7B model with PEFT, this notebook is run on Google Colab with an A100. If you're working with a smaller machine, I'd encourage to try the 2B model instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Time: 25 min | Level: Beginner | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1-CPpToBFPTqIYohxERhRFEU5FNP_vgbQ?usp=sharing)   |\n",
    "| --- | ----------- | ----------- |\n",
    "| Author | [Nirant Kasliwal](https://nirantk.com/about/) | "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "rOTVBRFAj2U-"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    }
   ],
   "source": [
    "!pip install -U fastembed datasets qdrant-client peft transformers accelerate bitsandbytes -qq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-03-30T00:45:24.814968Z",
     "start_time": "2024-03-30T00:45:24.811138Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aAfdPG15j2U_",
    "outputId": "c457a5dd-bccb-4b56-8c7f-878dfc94884d"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from datasets import load_dataset\n",
    "from peft import AutoPeftModelForCausalLM\n",
    "from qdrant_client import QdrantClient\n",
    "from qdrant_client.models import PointStruct, VectorParams, Distance\n",
    "from transformers import AutoTokenizer\n",
    "\n",
    "from fastembed import TextEmbedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf_token = \"<YOUR_HF_TOKEN_HERE>\"  # Get your token from https://huggingface.co/settings/token, needed for Gemma weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T4f74a_gj2U_"
   },
   "source": [
    "## Setting Up\n",
    "\n",
    "We'll download the dataset, our LLM model weights and embedding model weights next"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1xh1P9z5kAxd"
   },
   "outputs": [],
   "source": [
    "embedding_model = \"sentence-transformers/paraphrase-multilingual-mpnet-base-v2\"\n",
    "model_id = \"Telugu-LLM-Labs/Indic-gemma-2b-finetuned-sft-Navarasa\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CpCX7dOlj2VA"
   },
   "outputs": [],
   "source": [
    "ds = load_dataset(\"nirantk/chaii-hindi-and-tamil-question-answering\", split=\"train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "x-AkWw420GgW",
    "outputId": "460de1e2-aa4d-4dde-cf1d-abb38288bc97"
   },
   "outputs": [],
   "source": [
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wx1yWuc9zHwg"
   },
   "source": [
    "This dataset has questions and contexts which have corresponding answers. The answers must be found by the LLM. This is an extractive Question Answering problem.\n",
    "\n",
    "In order to do this, we'll setup an embedding model from FastEmbed. And then add it to Qdrant in memory mode, which is powered by Numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "8dbaa1a6fdcf4b4d805a94c8cce564eb",
      "4b89c0d412a64dfd80d82fd818977702",
      "a2a21109220d407095b8a3e916424365",
      "1d5b6d1ec1154c3ebaf98d62be90f967",
      "16d89c278bc34208ab76983db54e6465",
      "324c73c2bb464f919314f5a40a1e89e7",
      "b304cbb2fa554d909441b51c3a4682ae",
      "67ba7e82e98c4408b96bba4fe07e39a3",
      "9a63c76313274dd1b8aabf6470f9fa32",
      "11e7c0fb108d41cd938dada314157342",
      "f4539e2f491a47ae937b59911ea69d8e"
     ]
    },
    "id": "e1FzPDS6j2VA",
    "outputId": "3d69a566-0c67-45f9-b768-e0c498863d95"
   },
   "outputs": [],
   "source": [
    "embedding_model = TextEmbedding(model_name=embedding_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vg2WgC7_zpfv"
   },
   "source": [
    "We'll use the 7B model here, the 2B model isn't great and was suffering from reading comprehension challenges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the Navarasa LLM\n",
    "\n",
    "We'll download the Navarasa LLM from TeluguLLM-Labs. This is a 7B model with PEFT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "d5a4726255f24bef914841e541a03a30",
      "f1371ea81d6e4930ad3712e370bf1299",
      "6e5278269f3c4723be26a03baf12d613",
      "27cc073f21404a38b58db04974458b5e",
      "503d80fa1c3d4e1db0fbc8b2ab3004a5",
      "9e4a0a7cb9bc4132b562691e657dc5ab",
      "7a255a55ad3b4cc49470285073addaba",
      "cefda21f046e444a8168129e3dc869af",
      "15548e2974c749659bc2b8502906a8e4",
      "296e600afb144a298d3e2223973e58f9",
      "7cfa648b10c2430d974ecc053d823958"
     ]
    },
    "id": "zst575Z6j2VA",
    "outputId": "9237546c-7ca0-479b-9cf9-4396ccc15dc7"
   },
   "outputs": [],
   "source": [
    "model = AutoPeftModelForCausalLM.from_pretrained(\n",
    "    model_id,\n",
    "    load_in_4bit=False,\n",
    "    token=hf_token,\n",
    ")\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LWVHPXIFzzgW"
   },
   "source": [
    "## Embed the Context into Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions, contexts = list(ds[\"question\"]), list(ds[\"context\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5yjBVH3Yj2VA"
   },
   "outputs": [],
   "source": [
    "context_embeddings: list[np.ndarray] = list(\n",
    "    embedding_model.embed(contexts)\n",
    ")  # Note the list() call - this is a generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "X3qnhlyll3dT",
    "outputId": "b496f5b4-219a-45fe-8df4-8869d9c6c4e0"
   },
   "outputs": [],
   "source": [
    "len(context_embeddings[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Sbe7HN6Qmnu7"
   },
   "outputs": [],
   "source": [
    "def embed_text(text: str) -> np.array:\n",
    "    return list(embedding_model.embed(text))[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lIxvYWiQj2VA"
   },
   "outputs": [],
   "source": [
    "context_points = [\n",
    "    PointStruct(id=idx, vector=emb, payload={\"text\": text})\n",
    "    for idx, (emb, text) in enumerate(zip(context_embeddings, contexts))\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QvuRKZCSmJTc",
    "outputId": "bd528f49-b3af-4da5-9a80-124b8ce93b80"
   },
   "outputs": [],
   "source": [
    "len(context_points[0].vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nzwBvJ59zxAc"
   },
   "source": [
    "## Insert into Qdrant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TkfZxG1gkQ_k",
    "outputId": "5c190152-ee40-487c-8aef-858cd5bdccf9"
   },
   "outputs": [],
   "source": [
    "search_client = QdrantClient(\":memory:\")\n",
    "\n",
    "search_client.create_collection(\n",
    "    collection_name=\"hindi_tamil_contexts\",\n",
    "    vectors_config=VectorParams(size=len(context_points[0].vector), distance=Distance.COSINE),\n",
    ")\n",
    "search_client.upsert(collection_name=\"hindi_tamil_contexts\", points=context_points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting a Question\n",
    "\n",
    "I've randomly selected a question here, with a specific and we then find the answer to it. We have the correct answer for it too -- so we can compare the two when you run the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ORNuYzuOm2tH",
    "outputId": "bddb668c-434f-43ad-f773-9bbdc5383612"
   },
   "outputs": [],
   "source": [
    "idx = 997\n",
    "\n",
    "question = questions[idx]\n",
    "print(question)\n",
    "search_context = search_client.search(\n",
    "    query_vector=embed_text(question), collection_name=\"hindi_tamil_contexts\", limit=2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F5TDUjpTvtnX",
    "outputId": "1105b324-b698-4b07-9af5-12394e69fb6e"
   },
   "outputs": [],
   "source": [
    "search_context_text = search_context[0].payload[\"text\"]\n",
    "len(search_context_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the Model with a Question & Context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FwT2njapj2VA"
   },
   "outputs": [],
   "source": [
    "input_prompt = \"\"\"\n",
    "Answer the following question based on the context given after it in the same language as the question:\n",
    "### Question:\n",
    "{}\n",
    "\n",
    "### Context:\n",
    "{}\n",
    "\n",
    "### Answer:\n",
    "{}\"\"\"\n",
    "\n",
    "input_text = input_prompt.format(\n",
    "    questions[idx],  # question\n",
    "    search_context_text[:2000],  # context\n",
    "    \"\",  # output - leave this blank for generation!\n",
    ")\n",
    "\n",
    "inputs = tokenizer([input_text], return_tensors=\"pt\")\n",
    "\n",
    "outputs = model.generate(**inputs, max_new_tokens=50, use_cache=True)\n",
    "response = tokenizer.batch_decode(outputs)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "jXw3aILgvMdp",
    "outputId": "fe73260c-26c4-49c9-819b-24aa50c090ad"
   },
   "outputs": [],
   "source": [
    "response.split(sep=\"### Answer:\")[-1].strip(\"<eos>\").strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "DKeqAZEm0C0u",
    "outputId": "44780197-c7fa-496e-e166-3512321f527d"
   },
   "outputs": [],
   "source": [
    "ds[idx][\"answer_text\"]"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "11e7c0fb108d41cd938dada314157342": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "15548e2974c749659bc2b8502906a8e4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "16d89c278bc34208ab76983db54e6465": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1d5b6d1ec1154c3ebaf98d62be90f967": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_11e7c0fb108d41cd938dada314157342",
      "placeholder": "​",
      "style": "IPY_MODEL_f4539e2f491a47ae937b59911ea69d8e",
      "value": " 9/9 [00:00&lt;00:00, 757.75it/s]"
     }
    },
    "27cc073f21404a38b58db04974458b5e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_296e600afb144a298d3e2223973e58f9",
      "placeholder": "​",
      "style": "IPY_MODEL_7cfa648b10c2430d974ecc053d823958",
      "value": " 4/4 [00:50&lt;00:00, 10.44s/it]"
     }
    },
    "296e600afb144a298d3e2223973e58f9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "324c73c2bb464f919314f5a40a1e89e7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b89c0d412a64dfd80d82fd818977702": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_324c73c2bb464f919314f5a40a1e89e7",
      "placeholder": "​",
      "style": "IPY_MODEL_b304cbb2fa554d909441b51c3a4682ae",
      "value": "Fetching 9 files: 100%"
     }
    },
    "503d80fa1c3d4e1db0fbc8b2ab3004a5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "67ba7e82e98c4408b96bba4fe07e39a3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6e5278269f3c4723be26a03baf12d613": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cefda21f046e444a8168129e3dc869af",
      "max": 4,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_15548e2974c749659bc2b8502906a8e4",
      "value": 4
     }
    },
    "7a255a55ad3b4cc49470285073addaba": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7cfa648b10c2430d974ecc053d823958": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8dbaa1a6fdcf4b4d805a94c8cce564eb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4b89c0d412a64dfd80d82fd818977702",
       "IPY_MODEL_a2a21109220d407095b8a3e916424365",
       "IPY_MODEL_1d5b6d1ec1154c3ebaf98d62be90f967"
      ],
      "layout": "IPY_MODEL_16d89c278bc34208ab76983db54e6465"
     }
    },
    "9a63c76313274dd1b8aabf6470f9fa32": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "9e4a0a7cb9bc4132b562691e657dc5ab": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a2a21109220d407095b8a3e916424365": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_67ba7e82e98c4408b96bba4fe07e39a3",
      "max": 9,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9a63c76313274dd1b8aabf6470f9fa32",
      "value": 9
     }
    },
    "b304cbb2fa554d909441b51c3a4682ae": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cefda21f046e444a8168129e3dc869af": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d5a4726255f24bef914841e541a03a30": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f1371ea81d6e4930ad3712e370bf1299",
       "IPY_MODEL_6e5278269f3c4723be26a03baf12d613",
       "IPY_MODEL_27cc073f21404a38b58db04974458b5e"
      ],
      "layout": "IPY_MODEL_503d80fa1c3d4e1db0fbc8b2ab3004a5"
     }
    },
    "f1371ea81d6e4930ad3712e370bf1299": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9e4a0a7cb9bc4132b562691e657dc5ab",
      "placeholder": "​",
      "style": "IPY_MODEL_7a255a55ad3b4cc49470285073addaba",
      "value": "Loading checkpoint shards: 100%"
     }
    },
    "f4539e2f491a47ae937b59911ea69d8e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
